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(57) Abstract: A method and apparatus are disclosed for generating a user profile in a recommendation system based on the current 
mood of the user. The present invention associates each session, such as a viewing session, with one or more current moods of the 
user The present invention learns the users preferences in accordance with various moods T and utilizes such mood-based viewing 
preferences to generate corresponding recommendations. In one implementation, an electronic programming guide is provided that 
allows a viewer to select one or more programs that the viewer is likely to find attractive, based on his or her current mood. 
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Method and apparatus for generating recommendations based on current mood of user 



Field of the Invention 

The present invention relates to recommenders, such as recommenders for 
television programming or other content, and more particularly, to a method and apparatus 
for making recommendations, such as recommendations of television programs or other 
5 content, based on the current mood of the user 



Background of the Invention 

The number of media options available to individuals is increasing at an 
exponential pace. As the number of channels available to television viewers has increased, 

10 for example, along with the diversity of the programming content available on such channels, 
it has become increasingly challenging for television viewers to identify television programs 
of interest. Historically, television viewers identified television programs of interest by 
analyzing printed television program guides, Typically, such printed television program 
guides contained grids listing the available television programs by time and date, channel and 

15 title. As the number of television programs has increased, it has become increasingly 
difficult to effectively identify desirable television programs using such printed guides. 

More recently, television program guides have become available in an 
electronic format, often referred to as electronic program guides (EPGs). Like printed 
television program guides, EPGs contain grids listing the available television programs by 

20 time and date, channel and title. Some EPGs, however, allow television viewers to sort or 
search the available television programs in accordance with personalized preferences. In 
addition, EPGs allow for on-screen presentation of the available television programs. 

While EPGs allow viewers to identify desirable programs more efficiently 
than conventional printed guides, they suffer from a number of limitations, which if 

25 j overcome, could further enhance the ability of viewers to identify desirable programs. For 
example, many viewers have a particular preference towards, or bias against, certain 
categories of programming, such as action-based programs or sports programming. Thus, the 
viewer preferences can be applied to the EPG to obtain a set of recommended programs that 
may be of interest to a particular viewer. 
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Thus, a number of tools have been proposed or suggested for recommending 
television programming. The Tivo™ system, for example, commercially available from Tivo, 
Inc., of Sunnyvale, California, allows viewers to rate shows using a "Thumbs Up and 
Thumbs Down" feature and thereby indicate programs that the viewer likes and dislikes, 
5 respectively. In this manner, the Tivo™ system implicitly derives the viewer's preferences 
from previous television programs that the viewer liked or did not like. Thereafter, the Ti Vo 
receiver matches the recorded viewer preferences with received program data, such as an 
EPG, to make recommendations tailored to each viewer. 

Implicit television program recommenders generate television program 

10 recommendations based on information derived from the viewing history of the viewer, in a 
non-obtrusive manner. Explicit television program recommenders, on the other hand, 
explicitly question viewers about their preferences for program attributes, such as title, genre, 
actors, channel and date/time, to derive viewer profiles and generate recommendations. 

While such television program recommenders identify programs that are likely 

1 5 of interest to a given viewer, they suffer from a number of limitations, which if overcome, 
could further improve the quality of the generated program recommendations. For example, 
conventional tools for generating television program recommendations consider a person's 
viewing history as a whole when generating a viewer profile and television program 
recommendation scores. Thus, the identified programs have no particular correlation to the 

20 current interests or mood of the viewer. A need therefore exists for a method and apparatus 
for generating television program recommendations that is responsive to the current mood of 
the viewer. 

Summary of the Invention 
25 Generally, a method and apparatus are disclosed for generating a user profile 

in a recommendation system based on the current mood of the user. The present invention 
thus learns the user's preferences in accordance with various moods, and thereafter utilizes 
such mood-based preferences to generate recommendations that are tailored to the current 
mood of the user. 

30 The present invention detects the user's mood by processing audio or visual 

information, such as the facial expression of the user. Once the mood is detected, the 
behavior associated with a given session can be associated with the current moods of the 
viewer. In one implementation, the present invention provides an electronic programming 
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guide that allows a viewer to select one or more programs that the viewer is likely to find 
attractive, based on his or her current mood. 

A more complete understanding of the present invention-, as well as further 
features and advantages of the present invention, will be obtained by reference to the 
5 following detailed description and drawings. 



Brief Description of the Drawings 

FIG. 1 illustrates a television programming recommender in accordance with 
the present invention; 

10 FIG. 2 illustrates a sample table from the program database of FIG. 1 ; 

FIG. 3 A illustrates a sample table from a Bayesian implementation of an 
implicit viewer profile of FIG. 1; 

FIG. 3B illustrates a sample table from a viewing history used by a decision 
tree (DT) recommender; 
15 FIG, 3C illustrates a sample table from a viewer profile generated by a 

decision tree (DT) recommender from the viewing history of FIG, 3B; 

FIG, 4 is a flow chart describing an exemplary mood detection and profile 
update process embodying principles of the present invention; and 

FIG, 5 is a flow chart describing an exemplary mood-based recommendation 
20 process embodying principles of the present invention. 

Detailed Description 

FIG. 1 illustrates a television programming recommender 100 in accordance 
with the present invention* As shown in FIG. 1, the television programming recommender 

25 100 evaluates each of the programs in an electronic programming guide (EPG) 130 to 
identify programs of interest to one or more viewer(s) 140. The set of recommended 
programs can be presented to the viewer 140 using a set-top terminal/television 1 60, for 
example, using well known on-screen presentation techniques. While the present invention is 
illustrated herein in the context of television programming recommendations, the present 

30 invention can be applied to any automatically generated recommendations that are based on a 
behavior history, such as a viewing history or purchase history. 

According to one feature of the present invention, the television programming 
recommender 100 generates a user profile 300, discussed below in conjunction with FIGS. 
3 A and 3C, based on the current mood of the viewer, in addition to the more conventional 
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viewing behavior of the viewer. While a conventional recommender considers a person's 
viewing history as a whole when generating a viewer profile, the present invention treats the 
viewer's preferences as a multi-class problem, and associates each viewing session with one 
or more current moods of the viewer. Thus, the present invention learns the viewer's 
5 preferences in accordance with various moods, and utilizes such mood-based viewing 
preferences to generate program recommendations. In this manner, an electronic 
programming guide is provided that allows a viewer to select one or more programs that the 
viewer is likely to find attractive, based on his or her current mood. 

! - As shown in FIG, 1, the television programming recommender 100 includes 

10 one or more audio/visual capture devices 150-1 through 150-N (hereinafter, collectively 
referred to as audio/visual capture devices 150) that are focused on the viewer 140. The 
audio/visual capture devices 150 may include, for example, a pan-tilt-zoom (PTZ) camera for 
capturing video information or an array of microphones for capturing audio information, or 
both. 

15 The audio or video images (or both) generated by the audio/visual capture 

devices 150 are processed by the television programming recommender 100, in*a manner 
discussed below in conjunction with FIGS. 4 and 5, to identify one or more predefined 
moods of the viewer 140. As discussed below, facial expression processing techniques may 
be employed to analyze the face of the viewer to detect, for example, if the viewer is happy 

20 or sad- In addition, audio processing techniques may be employed to analyze sounds made 
by the viewer to detect, for example, laughing or crying, which may suggest the current mood 
of the viewer. The mood of the viewer may be detected, for example, when profile 
information is recorded, or when a recommendation is about to be generated (or both). 

As shown in FIG. 1, the television programming recommender 100 contains a 

25 program database 200, one or more viewer profiles 300, a mood detection and profile update 
process 400 and a mood-based recommendation process 500, each discussed further below in 
conjunction with FIGS. 2 through 5, respectively. Generally, the program database 200 
records information for each program that is available in a given time interval. One 
illustrative viewer profile 300, shown in FIG. 3 A, is an implicit viewer profile that is 

30 typically derived from the viewing history of the viewer, based on the set of programs that 
the viewer liked or disliked. Another exemplary viewer profile 300% shown in FIG. 3C, is 
generated by a decision tree recommender, based on an exemplary viewing history 360, 
shown in FIG. 3B. 
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The mood detection and profile update process 400 processes the video or still 
images (or both) generated by the audio/visual capture devices 150 to sense the current mood 
of the viewer and to learn the viewer's preferences when in such a mood. The mood-based 
recommendation process 500 utilizes the mood-based viewing preferences developed by the 
5 mood detection and profile update process 400 to generate program recommendations based 
on the derived current mood of the viewer* 

The television program recommender 100 may be embodied as any computing 
device, such as a personal computer or workstation, that contains a processor 120, such as a 
central processing unit (CPU), and memory 110, such as RAM and/or ROM. In addition, the 

10 television programming recommender 100 may be embodied as any available television 
program recommender, such as the Tivo™ system, commercially available from Tivo, Inc,, 
of Sunnyvale* California, or the television program recommenders described in United States 
Patent Application Serial No, 09/466,406, filed December 17, 1999, entitled "Method and 
Apparatus for Recommending Television Programming Using Decision Trees," (Attorney 

1 5 Docket No. 700772), United States Patent Application Serial No, 09/498,271, filed Feb. 4, 
2000, entitled "Bayesian TV Show Recommender," (Attorney Docket No. 700690) and 
United States Patent Application Serial No. 09/627,139, filed July 27, 2000, entitled "Three- 
Way Media Recommendation Method and System," (Attorney Docket No. 7009 13), or any 
combination thereof, as modified herein to carry out the features and functions of the present 

20 invention. 

FIG. 2 is a sample table from the program database 200 of FIG. 1 that records 
information for each program that is available in a given time interval. As shown in FIG. 2, 
the program database 200 contains a plurality of records, such as records 205 through 220, 
each associated with a given program. For each program, the program database 200 indicates 

25 the date/time and channel associated with the program in fields 240 and 245, respectively. In 
addition, the title, genre and actors for each program are identified in fields 250, 255 and 270, 
respectively. Additional well-known features (not shown), such as duration, and description 
of the program, can also be included in the program database 200, 

FIG. 3 A is a table illustrating an exemplary implicit viewer profile 300. As 

30 shown in FIG. 3, the implicit viewer profile 300 contains a plurality of records 305-3 13 each 
associated with a different program feature. In addition, for each feature set forth in column 
330, the implicit viewer profile 300 provides the corresponding positive counts in fields 335 
through 345, and negative counts in field 350. According to a feature of the present 
invention, a positive count is provided for each distinct mood that is detected by the 
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television programming recommender 100. The various positive counts indicate the number 
of times the viewer watched programs having each feature while in the corresponding mood. 
The negative counts indicate the number of times the viewer did not watch programs having 
each feature. 

5 For each positive and negative program example (i.e., programs watched and 

not watched), a number of program features are classified in the user profile 300. For 
example, if a given viewer watched a given sports program ten times on Channel 2 in the late 
afternoon, while in a happy mood, then the positive counts (happy) associated with these 
features in the implicit viewer profile 300 would be incremented by 10 in field 345, and the 
10 negative counts would be 0 (zero). Since the implicit viewing profile 300 is based on the 
user's viewing history, the data contained in the profile 300 is revised over time, as the 
viewing history grows. Alternatively, the implicit viewer profile 300 can be based on a 
generic or predefined profile, for example, selected for the user based on his or her 
demographics. 

1 5 FIG. 3B is a table illustrating an exemplary viewing history 360 that is 

maintained by a decision tree television recommender. As shown in FIG. 3B, the viewing 
history 360 contains a plurality of records 361-369 each associated with a different program. 
In addition, for each program, the viewing history 360 identifies various program features in 
fields 370-379. The values set forth in fields 370-379 may be typically obtained from the 

20 electronic program guide 130, It is noted that if the electronic program guide 130 does not 
specify a given feature for a given program, the value is specified in the viewing history 360 
using a "?". 

FIG. 3C is a table illustrating an exemplary viewer profile 300' that may be 
generated by a decision tree television recommender from the viewing history 360 set forth in 
25 FIG. 3B. As shown in FIG. 3C, the decision tree viewer profile 300 9 contains a plurality of 
records 381-384 each associated with a different rule specifying viewer preferences. In 
addition, for each rule identified in column 390, the viewer profile 300' identifies the 
conditions associated with the rule in field 391 and the corresponding recommendation in 
field 392. 

3 0 For a more detailed discussion of the generating of viewer profiles in a 

decision tree recommendation system, see, for example, United States Patent Application 
Serial No- 09/466,406, filed December 17, 1999, entitled "Method and Apparatus for 
Recommending Television Programming Using Decision Trees," (Attorney Docket No. 
700772), incorporated by reference above. 
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FIG. 4 is a flow chart describing an exemplary mood detection and profile 
update process 400. As shown in FIG. 4, the mood detection and profile update process 400 
initially performs a test during step 410 to determine if an event has occurred to trigger the 
updating of the viewer profile 300, such as the end of a program or the selection of a new 
5 program channel. If it is determined during step 41 0 that event has not occurred to trigger the 
updating of the viewer profile 300, then program control returns to step 410 until such an 
event is detected. 

If* however, it is determined during step 410 that an event has occurred to 
trigger the updating of the viewer profile 300, then the current mood(s) of the viewer 140 are 

10 detected during step 420 using known facial expression analysis techniques, such as those 
described in "Facial Analysis from Continuous Video with Application to Human-Computer 
Interface/' Ph.D. Dissertation, University of Illinois at Urbana-Champaign (1999); or 
Antonio Colmenarez et al., "A Probabilistic Framework for Embedded Face and Facial 
Expression Recognition," Proc. of the Int'l Conf. on Computer Vision and Pattern 

1 5 Recognition," Vol. I, 592-97, Fort Collins, Colorado (1999), each incorporated by reference 
herein. The intensity of the facial expression may be obtained, for example, in accordance 
with the techniques described in United States Patent Application Serial Number 09/705, 
666, filed November 3, 2000, entitled "Estimation of Facial Expression Intensity Using a Bi- 
Directional Star Topology Hidden Markov Model," (Attorney Docket No. 701253), assigned 

20 to the assignee of the present invention and incorporated by reference herein. Generally, 

facial expression analysis detect the viewer's face in the field of view of the camera included 
in the audio/visual capture devices 1 50, and identify the particular facial expression exhibited 
by the viewer 140, such as a smile or frown. The facial expression is used to derive the 
current mood of the viewer 140, 

25 A test is performed during step 425 to determine if the television programming 

recommender 100 is a Bayesian recommender or a decision tree (DT) recommender. If it is 
determined during step 425 that the television programming recommender 100 is a Bayesian 
recommender, then the positive counts corresponding to the current mood(s) of the viewer 
140 are updated in the viewer profile 300 during step 430 for the program features associated 

30 with the current program. In addition, the negative counts are optionally updated in the 
viewer profile 300 during step 430 for the program features associated with one or more 
randomly selected programs that are not watched. 

If, however, it is determined during step 425 that the television programming 
recommender 100 is a decision tree (DT) recommender, then the rules in the viewer profile 
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300' are filtered during step 450 to identify only those rules associated with the current 
mood. Thereafter, the remaining rules (after filtering) are further processed to identify the 
rules that are satisfied by the current program. The current program is then added to the 
identified rules during step 470, as follows: 

5 

VT _ _ ! 1 New Program _ „ 

New Score = Current Score + ■ x IndicatedStrength 

[Total # Programs Covered by Rule 

where, the strength can have a value of 7 for a happy mood, 1 for a sad mood and 3 for a 
neutral mood. Alternatively, the viewer profile 300' of FIG. 3C can be updated during step 
470 by adding the watched program to the viewing history 360 and rebuilding the profile 

10 300*. Thereafter, program control terminates, 

FIG. 5 is a flow chart describing the mood-based recommendation process 500 
embodying principles of the present invention. The mood-based recommendation process 
500 utilizes the mood-based viewing preferences developed by the mood detection and 
profile update process 400 to generate program recommendations based on the derived 

1 5 current mood of the viewer* 

As shown in FIG. 5, the mood-based recommendation process 500 initially 
obtains the electronic program guide (EPG) 130 during step 5 10 for the time period of 
interest* Thereafter^ the appropriate viewer profiles 300 are obtained for the viewer during 
step 515. The mood-based recommendation process 500 then derives the current mood of 

20 viewer during step 520 using the audio/visual capture devices 150, in the same manner 
described above for the mood detection and profile update process 400, 

A test is performed during step 525 to determine if the television programming 
recommender 100 is a Bayesian recommender or a decision tree (DT) recommender. If it is 
determined during step 525 that the television programming recommender 100 is a Bayesian 

25 recommender, then a recommendation score is calculated for each program using the feature 
counts only for the current mood(s) during step 530, 

If, however, it is determined during step 525 that the television programming 
recommender 100 is a decision tree (DT) recommender, then the rules in the viewer profile 
300' are filtered during step 540 to identify only those rules associated with the current 

30 mood. Thereafter, the remaining rules (after filtering) are applied to all the programs in the 
time period of interest during step 550, A score is retrieved for each program from field 392 
of the profile 300 s corresponding to the first satisfied rule in the ordered list of the profile 
300 s . 
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Finally, the user is presented with the calculated recommendation score for 

each program during step 570, before program control terminates. 

It is to be understood that the embodiments and variations shown and 

described herein are merely illustrative of the principles of this invention and that various 

» 

5 modifications may be implemented by those skilled in the art without departing from the 
scope and spirit of the invention. 
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CLAIMS: 



1 . A method for recommending one or more items, comprising the steps of: 
obtaining a list of available items (130); 

detennining a current mood of a user (140); and 

generating a recommendation score for at least one of said available items 
5 (130) based on said current mood. 

2. The method of claim 1, wherein said current mood is determined using facial 
expression processing techniques. 

3 . The method of claim 1 9 wherein said current mood is determined using audio 
and/or video processing techniques. 

4. The method of claim 1 , wherein said current mood is determined by querying 
said user (140), ' 

5. The method of claim 1, wherein said one or more items are programs, content 
or products. 

6. A method for generating a user profile (300) indicating preferences of a user 
20 (140), comprising the steps of: 

monitoring one or more items that are selected by said user (140); 
determining a current mood of a user (140) during said selection; and 
recording an indication of said current mood with said item selection in said 

profile. 

25 

7. The method of claim 6, wherein said user profile (300) is associated with a 
program content recommender (100). 
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8. The method of claim 6, wherein said step of recording an indication of said 

item selection further comprises the step of incrementing one or more positive feature counts 
associated with said item and said current mood. 

5 9. The method of claim 6, wherein said current mood is determined by querying 

said user (140)* 

10. The method of claim 6, wherein said one or more items are programs, content 
or products. 

10 

11. A system (100) for recommending one or more items, comprising: 
a memory (110) for storing computer readable code; and 

a processor (120) operatively coupled to said memory (1 1 0), said processor 
(120) configured to: 

15 obtain a list of available items (130); j 

determine a current mood of a user (140); and 

generate a recommendation score for at least one of said available items (130) 
based on said current mood. 

20 12. The system (1 00) of claim 1 1, wherein said current mood is determined using 

audio and/or video processing techniques. 

13. The system (1 00) of claim 1 1 , wherein said current mood is determined by 
querying said user (140), 

25 

14. A system (100) for generating a user profile (300) indicating preferences of a 
user (140), comprising; 

a memory (110) for storing computer readable code; and 
a processor (120) operatively coupled to said memory (110), said processor 
30 (1 20) configured to: 

monitor one or more items that are selected by said user (140); 
determine a current mood of a user (140) during said selection; and 
record an indication of said current mood with said item selection in said 

profile. 
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15. The system (100) of claim 14, wherein said processor (120) is farther 
configured to increment one or more positive feature counts associated with said item and 
said current mood. 

16. The system (100) of claim 14, wherein said one or more items are programs, 
content or products. 
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